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ABSTRACT 


Over  the  past  20  years,  there  have  been  many  advancements  in  the  process  of  weather. 
Specific  contributions  to  improvements  in  the  forecast  process  come  from  improvements 
in  numerical  model  forecast  guidance  and  the  timely  availability  of  many  new 
observational  data  types.  As  numerical  forecasts  guidance  improves,  it  has  become 
important  to  document  the  value  a  human  forecaster  adds  to  the  forecast  process. 
Because  of  collateral  duties  and  career  requirements,  United  States  Navy  forecasters 
often  find  it  difficult  to  become  proficient  at  forecasting.  In  this  study,  a  basic  forecast 
process  was  used  to  identify  the  skill  in  forecasts  of  ceiling  and  visibility  made  by  human 
forecasters  and  produced  by  a  statistical  modification  to  numerical  model  fields.  Terminal 
Aerodrome  Forecasts  (TAF)  were  collected  for  eight  military  air  stations  and  compared 
to  the  performance  of  the  Localized  Aviation  Model  Output  Statistics  Program  (LAMP). 
Hit  rates  and  critical  success  indices  were  used  to  identify  forecast  skill.  Using  various 
timelines  and  categorical  partitions,  all  results  showed  that  there  were  little  to  no 
statistical  differences  between  the  TAF  and  LAMP  in  2013.  Finally,  a  case  study  was 
examined  to  highlight  the  capability  of  probability  forecasting  as  an  improvement  to  the 
forecast  process. 
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I.  INTRODUCTION 


A.  MOTIVATION 

Over  time,  the  process  of  weather  forecasting  has  evolved  along  several  fronts. 
Initially,  a  weather  forecaster  had  access  to  a  relatively  small  amount  of  information  upon 
which  a  complete  analysis  of  the  current  weather  conditions  and  prediction  of  future 
conditions  were  to  be  based.  This  analysis  had  to  be  equally  valid  for  a  forecaster’s 
region  and  remote  regions.  Because  information  was  scarce  and  its  accuracy  may  have 
been  suspect,  the  analysis  was  often  based  on  the  forecaster’s  subjective  interpretation  of 
the  data  and  a  persistence  factor  based  on  an  earlier  interpretation  of  limited  data. 
Therefore,  accurate  forecasts  were  very  dependent  on  knowledge  that  the  forecaster 
possessed  with  respect  to  basic  meteorological  principles,  the  regional  locations,  the 
accuracy  of  information  being  provided. 

Today,  the  forecast  process  remains  largely  one  in  which  information  is 
continually  made  available  for  interpretation  by  the  forecaster.  However,  there  is  now  an 
often  overwhelming  amount  of  infonnation  available.  The  increase  in  information  is  due 
to  two  factors.  One  is  the  vast  number  of  products  made  available  to  provide  forecast 
aids.  These  may  be  output  from  extremely  complex  numerical  weather  prediction  (NWP) 
models,  statistical  models  that  modify  or  blend  observations  and  output  and  NWP 
models,  or  software  that  displays  observations  from  a  very  large  number  of  sources. 

The  second  factor  is  the  capability  to  deliver  large  amounts  of  infonnation  to  the 
forecast  in  a  timely  manner.  The  forecast  process  is  often  strictly  controlled  by  time  in 
which  a  number  of  forecast  products  must  be  delivered  to  a  variety  of  customers.  Today, 
the  electronic  transfer  and  display  of  products  can  be  accomplished  rapidly  to  meet  the 
constraints  of  the  forecast  release  schedule. 

The  value  of  humans  as  forecaster  in  the  loop  (FITL)  during  the  forecasting 
process  has  been  in  debate  since  the  advent  of  NWP  models  in  the  1960s  (Stuart  et  al. 
2006).  Like  their  civilian  counterparts,  Navy  forecasters  draw  from  several  numerical 
models,  forecasting  rules  of  thumb,  and  personal  knowledge  when  building  their  forecast. 
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In  the  early  1990s,  forecasters  started  the  forecasting  process  by  hand  drawing  a  surface 
analysis  and  identifying  bogus  observations  to  be  deleted  from  the  model  run.  Today,  the 
numerical  model  is  operating  mostly  autonomously  and  arguably  has  become  the  primary 
consideration  of  today’s  Navy  forecasters. 

Over  the  past  20  years,  the  accuracy  of  NWP  forecasts  has  improved  considerably 
(Figure  1)  Here,  forecast  skill  is  expressed  in  tenns  of  the  anomaly  correlation  (AC), 
which  is  a  measure  of  forecast  skill  score  relative  to  climatology.  The  AC  of  the  five  day 
forecasts  from  the  Global  Forecast  System  (GFS)  from  the  National  Centers  for 
Environmental  Prediction  (NCEP)  has  improved  from  0.7  in  1990  to  about  0.88  in  2013. 
Using  the  AC  as  a  benchmark  for  forecast  skill  relative  to  climatology,  the  value  of  0.6  is 
the  acceptable  score  for  a  “useful.”  Figure  2  quantifies  that  improvement  in  the  terms  of 
days.  From  1990  to  2013  the  GFS  has  increased  its  useful  forecasts  of  500  hPa  height 
from  six  to  over  eight  days. 

In  addition  to  the  model  guidance,  there  are  several  advances  in  technology  like 
satellite  data  that  have  increased  resolution  and  frequency,  increased  access  to 
observations,  and  satellite  derived  winds  and  rain  rate  products.  All  of  these  and  much 
more  aid  in  providing  better  initial  conditions  upon  which  a  model  forecast  suite  is  based, 
and  they  aid  in  the  interpretation  of  model  data  by  the  forecaster.  Add  in  the  Internet  for 
the  rapid  delivery  of  all  these  products  and  it  is  no  wonder  that  forecasting  desk  has 
dramatically  changed  over  the  past  fifteen  years. 

As  late  as  1993,  the  Naval  Operational  Global  Atmospheric  Prediction  System 
(NOGAPS)  model  was  transmitted  to  the  forecaster  at  the  Naval  Western  Oceanography 
Center  twice  a  day  via  the  Naval  Environmental  Data  Network  (NEDN).  The  charts  were 
pulled  and  spliced  together  by  meteorological  technicians  and  hand  delivered  to  the 
forecaster’s  workstation.  Today,  Naval  Global  Environmental  Model  (NAVGEM), 
Coupled  Ocean  Atmosphere  Mesoscale  Prediction  System  (COAMPS),  GFS,  and  a  host 
of  other  global  and  regional  models,  are  delivered  via  the  Internet  to  provide  a  variety  of 
products  directly  to  the  forecaster’s  computer  in  almost  any  fonnat  at  least  four  times  a 
day. 
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Figure  1 .  Annually-averaged  anomaly  correlation  for  the  five  day  forecast  of 
500-hPa  heights  averaged  over  the  Northern  Hemisphere  (after  Yang 
2013) 
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Figure  2.  Annually-averaged  forecast  day  at  which  the  500  hPa  height  forecast 
loses  useful  skill  (AC=0.6)  for  the  Northern  Hemisphere  (after  Yang 
2013) 
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Even  with  this  significant  improvement  in  model  guidance  and  technology 
associated  with  observations  and  delivery  of  guidance,  it  is  still  widely  known  that  NWP 
does  not  provide  completely  accurate  analyses  and  forecasts,  and  are  only  valuable  in 
guiding  the  human  forecaster  in  the  prediction  process  under  some  uncertainty  (Stuart  et 
al.  2006).  Even  so,  it  has  been  difficult  to  quantify  exactly  how  much  the  forecaster 
improves  the  process  on  a  daily  basis.  This  is  partly  because  the  level  of  improvement  is 
highly  dependent  on  skill  level  and  dedication  of  the  particular  forecaster  (Doswell 
1986).  While  skill  level  can  be  increased  through  training  and  experience,  factors  like 
dedication,  motivation,  initiative  and  other  intangible  characteristics  are  under  the  control 
of  the  forecaster  alone.  Unfortunately,  these  characteristics  are  difficult  for  the  Navy  to 
specifically  seek  since  there  is  not  a  traditional  interview  process  when  selecting  entry- 
level  forecasters.  Additionally,  the  Navy  career  comes  with  multiple  responsibilities  that 
may  be  important  for  individual  career  advancement  but  may  be  counterproductive  to 
developing  a  true  professional  forecaster.  Rotating  through  various  duty  stations  every 
two  to  four  years  and  additional  duties  such  as  driving  unmanned  vehicles  and 
perfonning  hydrographic  surveys  help  to  slow  down  the  experience  process.  This  is  not 
to  say  that  there  are  not  extremely  talented  forecasters  in  the  Navy.  There  are  many 
forecasters  who  consistently  outperform  the  average. 

When  examining  the  training  and  development  of  Navy  forecasters,  there  are 
additional  factors  that  stand  out  as  areas  of  concern  in  regards  to  properly  grooming 
skillful  forecasters.  No  longer  do  forecasters  start  their  apprenticeship  by  obtaining 
weather  observations,  analyzing  the  current  state  of  the  atmosphere  with  hand-drawn 
charts,  or  take  upper-air  soundings  to  analyze  the  air  column  using  the  skew-T  diagram. 
The  lack  of  these  types  or  practical  experiences  combines  with  the  requirements  for 
collateral  duties  as  defined  above  to  prevent  the  forecaster  from  obtaining  valuable 
experience. 

Additional  factors  have  impacted  the  Navy  weather  forecast  process.  There  has 
been  a  the  realignment  of  the  meteorology  and  oceanography  (METOC)  community  into 
specialized  directorates.  Civilian  forecasters  now  sit  at  the  forecast  desk  of  all  Naval  Air 
Stations  (NAS)  and  several  key  maritime  forecasting  desks  at  the  Fleet  Weather  Centers 
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(FWC).  Both  of  these  factors  have  led  to  a  reduction  in  the  hands-on  forecasting 
experience  for  the  typical  Navy  forecaster  to  a  level  that  is  far  less  than  what  it  was 
fifteen  to  twenty  years  ago.  With  all  these  limiting  factors  to  gaining  meaningful 
experience,  it  would  be  expected  that  the  Navy  forecaster  is  leaned  heavily  on  much 
improved  model  guidance.  Lastly,  despite  the  advances  in  the  skill  level  of  the  model  and 
the  broadening  of  the  expected  technical  expertise,  the  baseline  forecasting  training  has 
yet  to  be  significantly  updated  to  keep  pace  with  modernization  of  the  forecasting 
positon.  The  Training  Course  Control  Document  for  Aerographer’s  Mate  Class  C-l 
Course,  dated  June  2013  is  still  very  much  the  same  forecasting  course  it  was  in  1996. 

The  goal  of  this  thesis  is  to  simply  compare  Navy  forecaster  perfonnance  to  the 
perfonnance  of  a  numerical  model  to  detennine  the  level  of  value  being  added  to  the 
NWP  guidance.  Beyond  assessing  value,  a  specific  forecast  application  is  considered  to 
define  how  frequent  forecast  input  would  be  critical  to  a  daily  operation.  This  provides  a 
measure  of  the  relative  frequency  by  which  a  Navy  forecaster  input  would  be  a  critical 
part  of  the  decision-making  process. 

B.  IMPORTANCE  TO  NAVAL  APPLICATION 

In  ever-increasing  budget  constraints  and  manning  reductions,  there  is  a 
requirement  to  seek  efficiencies  in  the  forecast  process.  These  include  improved 
techniques,  relevant  training,  and/or  automation  where  feasible.  Once  the  level  of  value 
added  to  a  NWP  guidance  model  by  the  Navy  forecaster  is  assessed,  better  decisions  can 
be  made  as  to  how  to  use  resources  to  capitalize  on  that  value.  Additionally,  over  a  broad 
range  of  forecasts  that  are  assessed,  it  may  be  a  relatively  small  number  that  have  direct 
impact  on  the  decision-making  process.  Whether  the  value  is  minimal  or  indispensable, 
better  decisions  can  be  made  to  affect  the  training,  equipping  and/or  the  placement  of 
Navy  forecasters  throughout  the  Navy’s  operations.  While  this  study  will  only  look  into  a 
small  portion  of  the  Navy  forecaster’s  responsibility,  it  should  be  considered  a  reliable 
indication  of  overall  forecasting  perfonnance. 
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C.  BACKGROUND 

1.  Terminal  Aerodrome  Forecast 

To  assess  the  FITL,  a  suitable  forecast  process  is  required.  In  terms  of  this  thesis, 
a  suitable  process  is  one  in  which  several  factors  can  be  examined.  These  are  defined  as: 

1)  The  forecast  is  produced  over  a  set  operational  timeline  and  schedule; 

2)  There  are  equivalent  model-based  and  forecaster-based  forecast  products; 

3)  There  is  an  identifiable  mission-critical  role  in  which  the  forecast  is  a  critical 
part  of  the  decision-making  process. 

Based  on  these  criteria,  production  of  the  Tenninal  Aerodrome  Forecast  (TAF)  is  chosen 
as  the  forecast  process  to  be  examined  in  this  thesis. 

Although  weather  forecasts  are  often  provided  out  to  five  days,  the  Navy 
forecaster’s  operational  impact  is  generally  centered  over  the  next  24  h  forecast  period. 
Specifically,  naval  airfields  rely  on  the  production  of  the  TAF  that  is  produced  for  a  24  h 
period  and  updated  at  least  every  eight  hours.  The  TAF  is  the  only  operational  forecast  in 
which  format,  transmission  time,  and  content  are  standardized  across  the  Navy  and 
produced  specifically  for  a  sole  purpose  of  safety  of  flight.  Other  operational  forecasts  are 
tailored  to  individual  customers  based  on  their  specific  mission  and  assets.  Of  all  types  of 
forecasts,  TAFs  are  required  to  be  most  precise  in  terms  of  measurable  weather 
conditions  and  in  tenns  of  timing  of  those  elements  (Riordan  and  Hansen  2002). 

Governed  by  Naval  Meteorology  and  Oceanography  Command  instruction 
3143.1H,  the  TAF  is  developed  by  forecasters  for  individual  airfields  to  specifically 
support  safety  of  flight  for  in  and  outbound  flights.  The  TAF  is  issued  every  eight  hours 
and  provides  expected  weather  conditions  for  the  next  24  hours  for  that  particular  airfield. 
It  is  considered  a  micro-scale  forecast  and  requires  precise  techniques  and  tenninology.  It 
is  also  used  locally  to  make  critical  base  operation  decisions,  such  as  fueling  and 
movement  of  ammunition.  The  TAF  addresses  predominate  and  temporary  forecast 
conditions.  The  predominate  forecasts  are  expected  to  occur  during  a  majority  of  the 
forecast  period,  whereas  the  temporary  conditions  are  expected  to  occur  less  than  an  hour 


at  a  time  and  less  than  half  the  forecast  period. 
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2.  Localized  Aviation  Model  Output  Statistics  Program  (LAMP) 

The  LAMP  is  a  system  of  objective  analyses,  simple  models,  regression 
equations,  and  related  thresholds  which  together  provide  guidance  for  sensible  weather 
forecasts  (Ghirardelli  2005).  Issued  hourly,  the  LAMP  updates  GFS-Model  output 
statistics  (MOS)  using  the  most  recent  observations.  The  GFS-MOS  are  station  specific 
text  bulletins  produced  every  six  hours  to  be  used  as  forecasting  guidance.  It  uses  the 
GFS  model  as  a  baseline  and  improves  on  that  using  observational  and  geoclimatic  data 
specific  to  the  individual  station  (Antolik  2012).  The  LAMP  is  run  on  NCEP  computers 
and  disseminated  centrally  from  NCEP.  Lamp  guidance  is  provided  for  over  1500 
stations  as  well  as  thunderstonn  guidance  on  a  20  km  grid  out  to  25  hours.  The  LAMP  is 
primarily  produced  and  distributed  as  an  aviation  forecasting  reference  model. 
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Figure  3.  A  sample  LAMP  Bulletin.  Available  from  the  National  Weather 
Service  at  http://www.nws.noaa.gov/mdl/gfslamp/docs 
/LAMP_description.shtml. 
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In  a  LAMP  bulletin  (Figure  3),  each  forecast  hour  is  presented  across  the  columns 
from  left  to  right.  The  meteorological  elements  that  are  verified  in  this  thesis  are  cloud 
cover  (CLD),  primary  ceiling  heights  (CIG),  conditional  ceiling  heights  (CCG),  primary 
visibility  (VIS)  and  conditional  visibility  (CVS). 

The  production  of  LAMP  bulletins  as  hourly  updates  to  the  six  hourly  GFS-MOS 
bulletins  is  diagramed  in  Figure  4.  For  example,  the  0600  UTC  GFS-MOS  bulletin  is 
issued  at  approximately  1000  UTC  since  there  is  an  approximate  4  h  processing  time. 
The  LAMP  bulletins  are  then  issued  hourly  based  on  the  1000  UTC  GFS-MOS  bulletin 
from  1000  UTC  through  1500  UTC.  After  1500  UTC,  the  1600  UTC  GFS-MOS  bulletin 
is  available,  which  is  then  the  basis  for  subsequent  LAMP  bulletins. 
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Figure  4.  Schedule  of  LAMP  updates  in  relation  to  the  GFS-MOS  bulletins 


The  LAMP  was  chosen  as  the  model  guidance  for  this  study  partly  because  of  its 
hourly  format  and  availability,  but  also  for  a  recent  verification  study  that  scored  LAMP 
better  than  a  number  of  regional  models.  In  this  study,  it  was  demonstrated  that  the 
LAMP  categorical  forecasts  of  ceiling  height  below  1000  ft  and  visibility  below  3  miles 
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are  more  accurate  than  the  20  km  Rapid  Update  Cycle  (RUC20)  and  the  Weather 
Research  and  Forecasting  Nonhydrostatic  Mesoscale  Model  (WRF-NMM)  (Rudack  and 
Ghirardelli  2010). 

The  Critical  Success  Index  (CSI)  scores  for  the  three  models  compared  in  the 
study  (Figures  5  and  6),  for  October  2006  through  March  2007,  compared  favorably  with 
respect  to  persistence  forecasting.  Persistence  can  be  defined  as  the  existence  of  statistical 
dependence  among  successive  occurrences  of  a  given  event  (Wilkes  1995).  The  CSI  is  a 
statistic  used  in  place  of  a  hit  rate  when  an  event  desired  occurs  significantly  less  than 
non-occurrence  (Wilkes  1995).  The  specific  format  of  the  CSI  is  defined  in  Chapter  II., 
but  the  CSI  accounts  for  correct  forecasts,  erroneous  forecasts  and  forecasts  of  events  that 
do  not  occur.  The  CSI  score  varies  from  zero  to  one,  with  one  being  a  perfect  score.  Since 
2006,  LAMP  has  undergone  a  number  of  improvements,  including  a  2012  upgrade  of  the 
ceiling  parameters  (McClung  2012)  and  has  also  benefited  from  the  continued 
improvement  of  the  GFS  model. 
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Figure  5.  The  CSI  for  ceiling  below  1000  ft  for  Oct  2006-Mar  2007  (Rudack 
and  Ghirardelli  2010) 


Projection  (hours) 

Figure  6.  The  CSI  for  visibility  below  3  nm  for  Oct  2006-Mar  2007  (Rudack 
and  Ghirardelli  2010) 
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II.  METHODOLOGY 


A.  DATA  SELECTION  AND  COLLECTION 

The  1500  UTC  TAF  and  LAMP  were  selected  as  the  primary  forecasts  to  be 
verified  due  to  their  availability  throughout  the  calendar  year  2013.  The  1500  UTC 
forecasts  are  also  issued  during  the  periods  of  active  runway  hours  across  the  continental 
United  States  and  should  represent  the  forecaster’s  best  effort.  The  TAFs  and 
observations  were  collected  from  a  weather  infonnation  service  provider  named 
OGIMET  (www.ogimet.com).  OGIMET  is  the  only  site  archiving  TAFs  electronically 
for  either  civilian  or  Department  of  Defense  forecasts.  The  OGIMET  database  begins  in 
2005  for  some  stations  and  Navy  TAFs  are  available  as  far  back  as  September  2006.  The 
LAMP  bulletins  were  attained  by  email  request  to  the  National  Weather  Service  Office  of 
Science  and  Technology.  Forecast  stations  (Figure  7)  were  selected  to  sample  various 
environmental  conditions  with  respect  to  continental  locations.  Naval  Air  Station  (NAS) 
Ocean  (KNTU)  and  NAS  Jacksonville  (KNIP)  were  selected  to  represent  the  eastern 
portion  of  North  America.  NAS  Ft.  Worth  (KNFW),  NAS  Meridian  (KNMM),  and  NAS 
Fallon  (KNFL)  were  selected  to  represent  the  continental  region.  Finally,  NAS  Lemoore 
(KNLC)  and  NAS  Whidbey  Is  (KNUW)  were  selected  to  represent  western  North 
America.  Additionally,  conditions  at  KNLC  represent  the  interior  central  valley  portion 
of  California  while  KNUW  is  a  Pacific  Northwest  coastal  location.  Wheeler  Army  Air 
Field  (AAF)  (PHHI)  in  Hawaii  was  selected  for  a  tropical/island  station. 
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NAS  Whidbey  Is. 


Figure  7.  Forecast  station  locations 

In  addition  to  the  full  24  hr  verification  in  2013,  this  study  separately  verified  the 
13-h  through  24  h  period  of  each  forecast  to  eliminate  the  nowcast.  The  intent  here  is  to 
provide  a  measure  of  forecaster  skill  several  hours  after  the  initial  observation. 

The  earliest  TAFs  available  from  archives  are  from  September  2006.  In  an  effort  to 
determine  the  rate  of  improvement  based  on  this  verification  style,  the  TAFs  from 
October  2006  through  March  2007  were  collected  and  compared  with  the  corresponding 
time  periods  in  2013.  Both  NAS  Lemoore  and  AAF  Wheeler  were  not  included  in  this 
comparison  due  to  low  number  of  times  when  Instrument  Meteorological  Conditions 
(IMC)  occur. 

B.  INSTRUMENT  METEOROLOGICAL  CONDITIONS 

As  the  results  will  later  show,  IMC  occurs  far  more  infrequently  than  Visual 
Meteorological  Conditions  (VMC).  Therefore,  this  study  focused  on  verification  of  IMC 
weather  conditions.  The  IMC  conditions  include  broken  to  overcast  ceilings  below  1000 
ft  and  visibility  less  than  3  miles.  VMC  conditions  occur  when  ceilings  are  3000  ft  or 
above  and  visibility  is  5  miles  or  above.  Marginal  Visual  Meteorological  Conditions 
(MVMC)  occur  when  ceilings  are  greater  or  equal  to  1000  ft  but  less  than  3000  ft  and/or 
visibility  is  greater  than  or  equal  to  3  miles  but  less  than  5  miles.  The  terms  Instrument 
Flight  Rules  (IFR)  and  Visual  Flight  Rules  (VFR)  are  often  interchanged  with  IMC  and 
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VMC.  However,  the  latter  terms  correctly  identify  the  flight  rules  for  pilots  that  are 
dictated  by  the  meteorological  conditions.  IMC  puts  IFR  into  effect  while  VMC  and 
MVMC  put  VFR  into  effect. 

Outside  of  lightening,  IMC  conditions  have  the  most  significant  impact  on  flight 
operations  at  an  airfield.  Not  all  precipitation  events  produce  IMC  conditions,  and 
although  verifying  precipitation  events  could  be  a  valuable  tool  in  some  cases,  it  is  the 
lower  ceilings  and/or  visibility  that  the  precipitation  produces  that  cause  impacts  to  air 
operations.  The  forecast  process  for  ceilings  and  visibility  generally  takes  into  account 
most  weather  elements  represented  in  the  TAF  and  included  in  the  model.  Therefore,  the 
verification  of  IMC  events  will  give  a  more  than  fair  representation  of  forecasting  skill  as 
a  whole. 

C.  VERIFICATION  PROCESS 

Observations  for  each  station  are  transmitted  at  the  beginning  of  every  hour. 
Additionally,  intennediate  observations,  called  specials,  are  transmitted  when  certain 
weather  criteria  are  met.  For  this  study,  all  observations  for  each  hour  are  grouped 
together  and  the  lowest  ceiling  and  visibility  from  any  of  those  observations  were  used  to 
represent  the  observed  event.  Similarly,  in  situations  where  the  TAF  had  multiple 
conditions  represented  between  the  predominate  and  temporary  forecast  lines,  the  lower 
of  the  two  values  for  ceiling  and  visibility  were  used  for  verification.  The  LAMP  uses 
conditional  lines  for  its  temporary  forecasts.  Although  these  lines  are  conditional  on  the 
occurrence  of  the  forecasted  precipitation,  which  is  a  probabilistic  forecast,  the  lower 
ceiling  or  visibility  in  either  the  predominate  or  conditional  lines  were  used  as  the 
forecasted  event,  regardless  of  the  probability. 

To  parse  the  TAF  data,  the  Microsoft  .Net  Framework  4  program  was  used  to 
place  the  original  bulletins  into  a  Microsoft  Excel  workbook.  Excel  was  then  the  main 
tool  used  to  ingest  observations  and  parse  the  LAMP  data.  Excel  then  verified  both  the 
TAF  and  LAMP  to  the  observation  for  each  hour  of  the  day.  Multiple  formula  were  then 
used  to  collect  the  required  data  and  statistics  to  use  in  the  results  chapter  of  this  study. 
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D.  STATISTICS 

The  verification  of  forecast  IMC  and  forecast  VMC  conditions  lends  itself  to  a 
categorical  system  based  on  a  forecast  yes/no  verified  against  the  observed  yes/no.  As 
defined  above,  the  occurrence  of  IMC  conditions  is  the  event  of  interest.  Therefore,  the 
categorical  forecast  verification  is  based  on  a  hit  rate  with  respect  to  the  forecast  and 
occurrence  of  IMC  conditions. 

Using  a  2-by-2  contingency  table  (Figure  8),  there  are  two  conns  of  the  traditional 
hit  rate  that  will  be  utilized  in  this  study.  First,  the  traditional  hit  rate  for  forecasts  of  IMC 
conditions  is  defined  using  a/(a+c).  This  verifies  every  IMC  forecast  from  the  TAF  and 
the  LAMP  forecast  to  as  a  simple  percentage  of  the  observed  events  that  were  correctly 
forecast.  Secondly,  the  fraction  of  IMC  forecast  conditions  that  are  hit  is  defined  as 
a/(a+b).  This  hit  rate  defines  the  TAF  and  LAMP  hourly  IMC  forecast  skill  as  a  simple 
percentage  of  how  often  the  categorical  forecast  was  correct.  Because  IMC  conditions 
occur  infrequently  compared  to  VMC,  a  correctly  forecast  VMC  event  is  considered  as  a 
correct  null.  The  critical  success  index  (CSI)  or  threat  score  is  a  better  measure  of 
categorical  forecast  accuracy  when  the  number  of  correct  null  cases  is  large  compared  to 
the  event  of  interest.  Using  Figure  8,  the  CSI  is  calculated  as  a/(a+b+c).  Therefore,  no 
credit  is  provided  for  a  correct  null  forecast.  The  CSI  will  be  calculated  for  IMC  forecasts 
in  the  TAF  and  LAMP. 

Another  important  verification  measure  is  the  false  alarm  rate  (FAR),  which  is 
defined  using  Figure  8  as  b/(b+d).  It  can  be  argued  that  forecast  misses  are  a  far  greater 
problem  than  false  alarms  since  a  false  alarm  at  least  prepares  the  customer  better  than  a 
miss.  But,  if  false  alarms  occur  too  frequently  you  can  create  the  “Boy  who  cried  wolf’ 
situation,  where  the  customer  no  longer  takes  the  proper  precautions  for  any  IMC  forecast 
(Roulston  and  Smith  2003).  In  this  stud,  emphasis  is  placed  on  the  HR  and  CSI  as 
measures  of  relative  forecast  accuracy  for  the  forecaster-generated  TAF  and  the 
automated  LAMP. 
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Event  Forecast 

Event  Observed 

Yes  (IMC) 

No  (VMC) 

Forecast  Total 

Yes  (IMC) 

a  (Hit) 

b  (False  Alarm) 

a+b 

No  (VMC) 

c  (Miss) 

d  (Correct  Null) 

c+d 

Obs  Total 

a+c 

b+d 

a+b+c+d=n 

Figure  8.  A  schematic  categorical  forecast  verification  2-by-2  contingency  table 
where  the  event  of  interest  is  IMC  conditions. 
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III.  RESULTS 


A.  FREQUENCY  OF  IMC  OCCURRENCE 

1.  Observed  Ceiling  and  Visibility  Categories 

There  were  67,805  observation  hours  collected  for  the  eight  stations  in  2013.  Of 
those,  only  4,130  or  6%,  were  IMC  ceilings  (Figure  9)  and  only  2,718  or  4%,  were  IMC 
visibility  (Figure  10).  Therefore,  ceilings  and  visibility  conditions  had  minimal  impact  on 
flight  operations  94%  of  the  time. 


Figure  9.  Observed  ceiling  conditions  for  all  8  stations  that  occurred  during 
2013.  Conditions  are  defined  as  visual  meteorological  conditions 
(VMC),  marginal  visual  meteorological  conditions  (MVMC),  and 
instrument  meteorological  conditions  (IMC). 
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»  VMC 
-  MVFC 
■  IMC 


Figure  10.  Observed  visibility  conditions  for  all  8  stations  that  occurred  during 
2013.  Conditions  are  defined  as  visual  meteorological  conditions 
(VMC),  marginal  visual  meteorological  conditions  (MVMC),  and 
instrument  meteorological  conditions  (IMC). 


Of  the  67,805  observation  hours  collected,  63,463  were  covered  by  both  a  TAF 
and  a  LAMP  forecasts  to  be  used  in  this  study.  Using  IMC,  MVMC,  and  VMC  as  the 
three  separate  forecast  categories  to  verify,  the  TAF  forecast  accuracy  was  78%,  for  both 
ceiling  and  visibility,  while  the  LAMP  forecast  accuracy  was  81%  for  ceiling  and  79% 
for  visibility.  (Figure  1 1).  Therefore,  on  first  look,  there  is  no  statistical  difference 
between  the  overall  accuracy  of  both  forecast  systems.  Due  to  the  relationship  between 
IMC  and  IFR,  as  described  in  an  earlier  chapter  and  to  assist  in  focusing  on  IMC  forecast 
accuracy,  the  term  “VMC”  will  include  all  VMC  and  MVMC  observations  or  forecasts, 
unless  otherwise  noted. 

The  next  set  of  verifications  completed  determined  if  there  was  an  hour-to-hour 
correspondence  between  all  (i.e.,  IMC  and  VMC)  observed  conditions  and  all  forecast 
conditions  during  2013.  The  forecasts  were  verified  against  the  observations  to  detennine 
how  often  the  hourly  observed  conditions  were  actually  forested  correctly.  Of  those 
63,463  observations,  59,515  ceiling  observations  and  60,873  visibility  observations  were 
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VMC.  Using  the  hit  rate  formula  d/(b+d),  95%,  or  56,695  of  the  ceiling  observations,  and 
94%,  or  56,988  of  the  visibility  observations  were  accounted  for  by  the  TAF,  with  very 
similar  numbers,  96%  and  93%,  respectively,  for  the  LAMP  (Figure  12).  When  IMC 
observations  are  the  condition  of  interest,  the  hit  rate  is  defined  by  a/(a+c).  For 
verification  of  the  3,948  IMC  ceiling  observations,  both  the  TAF  and  LAMP  accounted 
for  only  47%  of  the  observed  IMC  ceiling  hours  and  43%  of  the  2,590  IMC  visibility 
hours  (Figure  13).  To  put  this  into  perspective,  this  left  more  than  half  the  operation 
impacting  ceilings  and  visibility  unforecasted  in  2013.  Again,  when  looking  at  the 
observations  and  corresponding  forecasts,  there  is  very  little  difference  in  hit  rates 
between  the  TAF  and  the  LAMP. 

From  the  characterizations  of  IMC  and  VMC  conditions  overall,  it  is  clear  that  the 
meteorological  input,  which  is  assumed  to  be  critical  during  IMC  conditions,  is  especially 
key  during  this  relatively  infrequent  period  of  time.  However,  it  is  during  these  times 
when  the  Navy  forecaster  must  be  prepared  to  assess  infrequent  environmental  conditions 
and  know  how  they  are  represented  in  all  of  their  guidance  products. 
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TAF  LAMP 


Figure  1 1 .  The  TAF  and  LAMP  hit  rates  (percent)  for  VMC-MVMC-IMC 
categorical  forecasts  for  all  stations  in  2013.  Hit  rates  for  ceiling 
forecasts  are  defined  by  red  bars  and  for  visibility  forecasts  by  blue 
bars 
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Figure  12.  The  TAF  and  LAMP  hit  rates  (percent)  for  VMC  observations  for  all 
stations  during  2013.  The  red  bars  define  hit  rates  for  ceiling  forecasts 
and  the  blue  bars  define  hit  rates  for  visibility  forecasts 
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TAF  LAMP 

Figure  13.  The  TAF  and  LAMP  hit  rates  (percent)  for  IMC  observations  for  all 
stations  in  2013.  Hit  rates  for  ceiling  forecasts  are  defined  by  red  bars 
and  for  visibility  forecasts  by  blue  bars 


20 


2. 


Observed  Ceiling  and  Visibility  Categories  by  Season 


The  statistics  defined  above  are  unconditional  accuracies  for  all  stations 
throughout  2013.  The  results  may  be  sensitive  to  season  and/or  station  location. 
Therefore,  the  distributions  of  these  two  characteristics  are  defined  prior  to  assessing  the 
forecast  verifications  relative  to  season  and  station. 

The  distribution  of  observed  conditions  by  each  quarter  of  the  year  is  provided  in 
Table  1.  During  the  summer  quarter  (JUL-SEP),  IMC  ceiling  conditions  occurred  only 
4%  of  the  total  time,  which  was  the  lowest  frequency  of  all  the  seasons.  The  fall 
transition  months  (OCT-DEC)  had  the  highest  occurrence  of  IMC  ceiling  conditions  with 
9%.  For  visibility,  IMC  conditions  occurred  with  the  highest  frequency  of  5%  during  the 
winter  (JAN-MAR)  months. 

As  expected,  the  winter  season  contains  the  greatest  occurrence  of  ceiling  and 
visibility  IMC  conditions.  Therefore,  forecaster  input  would  be  of  most  value  during  that 
season. 
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Table  1.  The  seasonal  frequency  of  meteorological  category 
occurrence  for  all  stations  during  2013 


JAN-MAR 

16790  OBS 

CEILINGS 

VISIBILITY 

VMC 

13604 

81% 

VMC 

15432 

92% 

MVMC 

1940 

12% 

MVMC 

465 

03% 

IMC 

1246 

07% 

IMC 

893 

05% 

APR-JUN 

16983  OBS 

CEILINGS 

VISIBILITY 

VMC 

14692 

87% 

VMC 

16443 

97% 

MVMC 

1651 

10% 

MVMC 

206 

01% 

IMC 

640 

03% 

IMC 

334 

02% 

JUL-SEP 

16983  OBS 

CEILINGS 

VISIBILITY 

VMC 

15286 

90% 

VMC 

16156 

95% 

MVMC 

971 

06% 

MVMC 

275 

02% 

IMC 

726 

04% 

IMC 

552 

03% 

OCT-DEC 

17049  OBS 

CEILINGS 

VISIBILITY 

VMC 

13735 

81% 

VMC 

16156 

95% 

MVMC 

1796 

10% 

MVMC 

341 

02% 

IMC 

1518 

09% 

IMC 

552 

03% 
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3.  Observed  Ceiling  and  Visibility  Categories  by  Station 

The  distribution  of  observed  conditions  by  station  is  provided  in  Table  2.  The 
lowest  occurrence  of  IMC  conditions  occurred  at  NAS  Lemoore  with  only  91  ceiling 
IMC  events  and  231  visibility  IMC  events.  Nearly  all  of  these  events  occurred  in  the 
winter  months.  The  maximum  occurrence  of  IMC  conditions  was  at  NAS  Whidbey 
Island.  Overall  Whidbey  experienced  1,122  ceiling  IMC  events  and  812  IMC  visibility 
events.  This  equaled  13%  and  9%  of  their  observational  hours,  respectively. 


Table  2.  Frequency  of  meteorological  category  by  station  during 
2013 


LEMOORE 

8431  OBS 

CEILING 

VISIBILITY 

VMC 

8256 

98% 

VMC 

7840 

93% 

MVMC 

84 

01% 

MVMC 

360 

04% 

IMC 

91 

01% 

IMC 

231 

03% 

WHEELER 

8431  OBS 

CEILING 

VISIBILITY 

VMC 

6751 

80% 

VMC 

8241 

98% 

MVMC 

1469 

17% 

MVMV 

84 

01% 

IMC 

211 

03% 

IMC 

106 

01% 

MERIDIAN 

8431  OBS 

CEILING 

VISIBILITY 

VMC 

6591 

78% 

VMC 

7588 

90% 

MVMC 

1841 

22% 

MVMC 

262 

03% 

IMC 

940 

11% 

IMC 

581 

07% 

JACKSONVILLE 

8431  OBS 

CEILING 

VISIBILITY 

VMC 

7308 

87% 

VMC 

8101 

96% 
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MVMC 

637 

08% 

MVMC 

115 

01% 

IMC 

486 

06% 

IMC 

215 

03% 

WHIDBEY  IS. 

8557 

CEILING 

VISIBILITY 

VMC 

5973 

70% 

VMC 

7520 

88% 

MVMC 

1462 

17% 

MVMC 

225 

03% 

IMC 

1122 

13% 

IMC 

812 

09% 

OCEANA 

8557  OBS 

CEILING 

VISIBILITY 

VMC 

7111 

83% 

VMC 

7958 

93% 

MVMC 

706 

08% 

MVMC 

207 

02% 

IMC 

740 

09% 

IMC 

392 

05% 

FT.  WORTH 

8536  OBS 

CEILING 

VISIBILITY 

VMC 

7164 

84% 

VMC 

8262 

97% 

MVMC 

1024 

12% 

MVMC 

124 

01% 

IMC 

348 

04% 

IMC 

150 

02% 

FALLON 

8431  OBS 

CEILING 

VISIBILITY 

VMC 

8163 

97% 

VMC 

8092 

96% 

MVMC 

76 

01% 

MVMC 

108 

01% 

IMC 

192 

02% 

IMC 

231 

03% 
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B. 


FORECAST  PRODUCTS  VERIFICATION 


1.  2013  Forecast  Verification. 

In  section  A.l,  The  hit  rates  were  based  on  total  observations  in  each  category, 
a/(a+c)  for  IMC  conditions  or  d/(b+d)  for  VMC  conditions.  In  this  section,  the  hit  rate,  or 
fraction  of  hits  formula  used  will  account  for  all  forecasts  issued,  and  are  defined  as 
a/(a+b)  for  IMC  forecasts  and  d/(c+d)  for  VMC  forecasts. 

Considering  that  the  VMC  conditions  occurred  94%  of  the  time,  and  that  the 
forecast  accuracy  for  VMC  is  above  96%  for  both  LAMP  and  TAF  (Table  3),  a 
categorical  forecast  of  VMC  will  be  considered  the  correct  null  forecast  for  this 
verification.  Because  of  the  dominance  of  the  correct  null  event,  the  focus  is  on  the 
verification  of  IMC  events. 


Table  3.  VMC  forecast  hit  rates  for  all  stations  during  2013. 


2013 

FORECASTS 

(c+d) 

TAF 

Hits 

CEILING 

56651 

96% 

58760 

VISIBILITY 

56639 

98% 

58059 

2013 

FORECASTS 

(c+d) 

LAMP 

Hits 

CEILING 

57238 

96% 

59315 

VISIBILITY 

56792 

97% 

58270 

2.  IMC  Forecast  Accuracy  during  2013 
a.  Forecast  IMC  Fraction  of  Hits 

For  a  forecast  of  IMC,  the  fraction  of  hits  is  defined  as  a/(a+b).  In  this  case,  the 

fraction  of  hits  for  both  the  TAF  and  the  LAMP  IMC  visibility  forecasts  were  near  22% 

for  all  stations  (Figure  14).  These  values  are  much  lower  than  the  IMC  and  VMC  hit 

rates  define  above  where  both  the  TAF  and  LAMP  hit  rates  were  43%  for  the  IMC 

category.  For  ceiling  forecasts  the  fraction  of  hits  for  the  TAF  ceiling  IMC  forecasts  was 
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40%  and  for  the  LAMP  ceiling  IMC  forecasts  it  was  45%  (Figure  14).  These  values  are 
closer  to  the  47%  observational  hit  rates  defined  above.  Although  the  LAMP  was  5% 
better  than  the  TAF  for  forecasts  of  IMC  ceiling  conditions  with  543  fewer  misses,  TAF 
and  LAMP  had  similar  False  Alarm  Ratios  (FAR)  of  53%. 

The  distribution  of  the  fraction  of  hits  by  season  (Table  4)  indicates  that  LAMP 
had  a  higher  fraction  of  hits  for  categorical  forecasts  of  IMC  ceiling  conditions  during 
winter,  spring,  and  summer  and  was  slightly  lower  than  TAF  in  the  fall  (Oct-Dee).  The 
fraction  of  hits  for  the  TAF  categorical  IMC  forecasts  of  visibility  were  higher  during  the 
fall  and  winter  months  (OCT-MAR). 


80 


■  I 


TAF  LAMP 

Figure  14.  The  fraction  of  hits  (percent)  for  IMC  categorical  forecasts  for  all 

stations  during  2013.  The  red  bars  define  hit  rates  for  ceiling  forecasts 
and  the  blue  bars  define  hit  rates  for  visibility  forecasts. 
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Table  4.  The  fraction  of  hits  for  IMC  categorical  forecasts  for  all 
stations  during  2013.  The  number  of  forecasts  verified  is 
listed  for  each  element  of  ceiling  or  visibility.  The  number 
of  correct  forecast  is  listed  in  the  middle  box. 


JAN-MAR 

FORECASTS 

TAF 

JAN-MAR 

FORECATS 

LAMP 

CEILING 

1499 

627 

42% 

CEILING 

1393 

671 

48% 

VISIBILITY 

1354 

408 

30% 

VISIBILITY 

1746 

450 

26% 

APR-JUN 

FORECASTS 

TAF 

APR-JUN 

FORECASTS 

LAMP 

CEILING 

947 

236 

25% 

CEILING 

657 

207 

32% 

VISIBILITY 

998 

126 

13% 

VISIBILITY 

555 

98 

18% 

JUL-SEP 

FORECASTS 

TAF 

JUL-SEP 

FORECASTS 

LAMP 

CEILING 

845 

319 

38% 

CEILING 

619 

299 

48% 

VISIBILITY 

1322 

195 

15% 

VISIBILITY 

1359 

193 

14% 

OCT-DEC 

FORECASTS 

TAF 

OCT-DEC 

FORECASTS 

LAMP 

CEILING 

1371 

660 

48% 

CEILING 

1479 

694 

47% 

VISIBILITY 

1257 

385 

31% 

VISIBILITY 

1533 

371 

24% 
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b.  Forecast  IMC  Critical  Success  Index 

As  defined  in  chapter  II,  CSI  is  a  measure  of  success  that  is  used  in  place  of  hit 
rate  when  the  event  desired  event  occurs  significantly  less  than  a  non-occurrence,  which 
is  considered  as  a  correct  null  forecast.  In  this  case,  VMC  occurs  at  a  far  greater 
frequency  than  IMC.  Although  hit  rate  only  takes  into  account  the  hits  and  misses  for  all 
events  and  categories,  CSI  also  takes  into  account  false  alarms. 

Table  5  and  6  define  the  CSI  scores  for  both  the  TAF  and  the  LAMP  for  ceiling 
and  visibility  forecasts.  As  the  hit  rates  and  fraction  of  hits  discussed  above  would 
suggest,  the  LAMP  has  a  higher  CSI  of  .30  versus  .27  for  the  TAF  since  LAMP  performs 
better  in  all  three  CSI  components;  hits,  misses,  false  alarms  (Table  5).  For  visibility 
forecasts,  the  TAF  marginally  outperforms  the  LAMP  in  all  three  components  for  CSI  as 
the  TAF  CSI  of .  18  is  only  slightly  higher  than  the  LAMP  value  of .  17  (Table  6). 


Table  5.  The  distribution  of  forecast  and  observed  ceiling  categories 
plus  the  CSI  for  each  case  during  2013. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

1842 

2109 

IRF 

1871 

2077 

VFR 

2820 

56651 

VFR 

2277 

57238 

CSI 

0.27 

CSI 

0.30 
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Table  6.  The  distribution  of  forecast  and  observed  visibility 
categories  plus  the  CSI  for  each  case  during  2013 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

1114 

1420 

IRF 

1112 

1478 

VFR 

3817 

56639 

VFR 

4081 

56792 

CSI 

0.18 

CSI 

0.17 

3.  Forecast  Accuracy  for  the  13-24  h  Forecast  Interval 

It  is  assumed  that  the  forecast  process  will  be  less  dependent  on  aids  and  more 
dependent  on  forecaster  experience  and  capability  as  the  forecast  interval  increases. 
Therefore,  the  second  12-h  period  is  verified  separately  from  the  first  0-12  h  forecast 
interval.  For  the  13-24  h  forecasts,  the  IMC  fraction  of  hits  (Table  7)  are  all  within  1%  of 
the  full  24  hr  data.  The  CSI  scores  all  decrease  at  a  rate  to  be  expected  as  the  forecast 
interval  increases  away  from  the  initiating  observation  (Tables  8  and  9).  For  ceiling,  the 
TAF  CSI  decreases  from  .27  to  .26  while  the  LAMP  CSI  decreases  from  .30  to  .28.  For 
visibility,  the  TAF  CSI  decreases  from  .18  to  .14,  while  the  LAMP  CSI  decreases  from 
.17  to  .16.  The  only  notable  change  is  that  the  LAMP  goes  from  slightly  underperforming 
the  TAF  in  visibility  forecast  for  the  24  hr  forecast,  to  outperforming  the  TAF  for  the  13- 
24  hours. 
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Table  7.  The  hit  rates  (percent)  for  IMC  categorical  forecasts  for  all 
stations  at  12-24  h  forecast  interval  during  2013.  The 
number  of  forecasts  verified  is  listed  for  each  element  of 
ceiling  or  visibility.  The  number  of  correct  forecast  is  listed 
in  the  middle  box. 


2013  13- 

24HR 

FORECASTS 

TAF 

LAMP 

CEILING 

2390 

987 

42% 

CEILING 

2390 

1064 

45% 

VISIBILITY 

1490 

502 

34% 

VISIBILITY 

1490 

601 

40% 

Table  8.  The  ceiling  forecast  CSI  for  forecast  interval  12-24  h  for  at 
all  stations  during  2013. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

987 

1398 

IRF 

1064 

1326 

VFR 

1484 

28094 

VFR 

1371 

28257 

CSI 

0.26 

CSI 

0.28 
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Table  9.  The  visibility  forecast  CSI  for  forecast  interval  12-24  h  for 
at  all  stations  during  2013 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

502 

1398 

IRF 

601 

889 

VFR 

1597 

28707 

VFR 

2182 

28346 

CSI 

0.14 

CSI 

0.16 

4.  Forecast  Accuracy  during  2006  and  2007 

For  this  set  of  data,  NAS  Lemoore  and  AAF  Wheeler  were  not  included  in  the 
2006-2007  or  2013  data  due  to  low  IMC  counts.  The  forecast  hit  rate  for  the  TAF  and 
LAMP  in  October  2006  through  March  2007  as  well  as  the  hit  rates  for  October  through 
December  2013  combined  with  January  through  March  2013  are  detailed  in  Tables  10 
and  1 1 .  The  improvements  in  both  forecast  systems  are  significant  with  the  TAF 
improving  at  a  much  higher  level  than  the  model.  For  2006-2007,  the  CSI  statistics 
(Tables  12  and  14)  are  both  much  lower  than  the  overall  2013  CSI  values  (Tables  5  and 
6).  But,  when  compared  to  the  same  quarters  of  October  through  December  2013,  and 
January  through  March  2013  (Tables  13  and  15),  the  difference  becomes  even  greater. 
During  these  two  comparable  quarters  in  2013,  the  TAF  ceiling  forecast  CSI  was 
improved  by  19%  and  the  LAMP  forecast  CSI  was  improved  by  17%  compared  to  the 
values  from  2006-2007. 

When  comparing  CSI  data,  the  TAF  improved  ceiling  forecasts  by  69%  compared 
to  the  66%  LAMP  improvement.  The  visibility  CSI  doubled  for  the  TAF  and  just  38% 
increase  for  the  LAMP  forecast.  The  improvement  in  the  models  is  obvious  when  looking 
at  Figures  1  and  2.  But,  the  greater  improvement  of  the  TAF  is  not  as  obvious.  One 
significant  change  is  that  during  2013,  the  1500  UTC  forecast  was  written  on-station  at 
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each  of  the  six  stations  verified.  From  2006  through  2012  these  forecasts  were  all  done 
via  reach  back  at  Fleet  Weather  Center  San  Diego  or  Norfolk. 


Table  10.  Te  fraction  of  hits  for  the  IMC  forecasts  during  October 
2006-March  2007. 


2006/7 

FORECASTS 

TAF 

LAMP 

CEILING 

1927 

525 

27% 

CEILING 

1549 

511 

33% 

VISIBILITY 

1551 

231 

15% 

VISIBILITY 

1661 

305 

18% 

Table  11.  The  fraction  of  hits  for  the  IMC  forecasts  during  Jan-Mar 
2013  and  Oct-Dee  2013. 


2013 

FORECASTS 

TAF 

LAMP 

CEILING 

2723 

1248 

46% 

CEILING 

2681 

1325 

49% 

VISIBILITY 

2304 

707 

30% 

VISIBILITY 

2585 

678 

26% 
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Table  12.  The  CSI  for  ceiling  forecasts  made  during  October  2006- 
March  2007. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

525 

902 

IRF 

511 

916 

VFR 

1402 

22677 

VFR 

1038 

22139 

CSI 

0.19 

CSI 

0.21 

Table  13.  The  CSI  for  ceiling  forecasts  during  the  period  of  Jan- 
March2013and  Oct-Dee  2013. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR 

Obs 

IFR 

1248 

1244 

IRF 

1325 

1167 

VFR 

1475 

20866 

VFR 

1356 

21446 

CSI 

0.31 

CSI 

0.34 

Table  14.  The  CSI  for  visibility  forecasts  made  during  October  2006 
-  March  2007. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

231 

461 

IRF 

305 

387 

VFR 

1320 

22493 

VFR 

1356 

22556 

CSI 

0.11 

CSI 

0.15 
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Table  15.  The  CSI  for  visibility  forecasts  during  the  combined  period 
of  Jan-March  2013  and  Oct-Dee  2013. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

707 

772 

IRF 

678 

772 

VFR 

1597 

21751 

VFR 

1907 

21751 

CSI 

0.23 

CSI 

0.20 

5.  2013  Non-VMC  Accuracy  “Saw  Something” 

The  verification  of  forecast  IMC  conditions  explained  above  paired  the  MVMC 
events  with  the  VMC  events.  This  gave  a  “yes”  or  “no”  verification  for  IMC  events. 
When  pairing  MVMC  with  IMC,  this  provides  a  “yes”  or  “no”  verification  of  below 
VMC  events,  which  expands  the  correct  category.  Essentially,  this  will  give  the  TAF  and 
the  LMAP  credit  when  the  forecast  and  the  observation  are  both  below  VMC.  This  is  not 
as  strict  as  verifying  IMC  alone  and  allows  credit  for  when  the  TAF  or  FAMP  essentially 
lead  a  forecast  to  be  one  of  marginal  conditions,  or  essentially  the  forecast  “saw 
something  coming,”  but  missed  the  actual  IMC  category.  This  Non-VMC  or  “saw 
something  coming”  verification  gives  credit  for  any  hit  below  3000  feet  ceilings  and  5 
miles  visibility.  For  example,  the  TAF  has  ceilings  at  2000  feet  and  the  observation  has 
900  feet.  This  is  a  missed  IMC  event,  but  when  pairing  the  MVMC  and  IMC  categories, 
this  would  be  a  Non-VMC  event  hit. 

When  verified  in  this  set  scenario,  both  the  TAF  and  the  FAMP  improve  over  the 
IMC  scores  significantly.  Once  again  though,  they  both  remained  statistically  equal.  The 
ceiling  forecast  hit  rate  was  67%  and  visibility  forecast  hit  rates  was  71%  (Figure  16). 
The  CSI  values  also  improved  for  ceiling  forecasts  as  the  TAF  CSI  was  .36  and  the 
FAMP  ceiling  forecast  CSI  was  .42  (Table  16).  The  visibility  forecast  CSI  however, 
remained  mostly  unchanged.  This  is  attributed  to  the  significantly  high  false  alarm  rate  in 
the  b  block  (Figure  8)  of  the  2-by-2  contingency  table  (Table  17).  This  indicates  that  both 
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the  LAMP  and  the  TAF  significantly  over  forecast  MVMC  conditions.  It  is  not  clear 
from  this  study  if  that  over  forecasting  is  in  the  predominate  forecast  lines,  or  the 
temporary  forecast  lines. 


TAF  LAMP 


Figure  15.  The  hit  rate  (percent)  for  forecast  Non-VMC  conditions  during  2013 
“saw  something  coming.”  The  red  bars  define  hit  rates  for  ceiling 
forecasts  and  the  blue  bars  define  hit  rates  for  visibility  forecasts 


Table  16.  The  CSI  for  non-VMC  ceiling  category  forecasts  during 
2013. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

6500 

8158 

IRF 

6594 

5805 

VFR 

3245 

45560 

VFR 

3151 

47913 

CSI 

0.36 

CSI 

0.42 
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Table  17.  The  CIS  for  non-VMC  visibility  category  forecasts  during 
2013. 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

2809 

11445 

IRF 

2801 

11057 

VFR 

1168 

48041 

VFR 

1176 

47913 

CSI 

0.18 

CSI 

0.18 

6.  Reach  Back  Forecasting  versus  On-Station  Forecasting 

Between  2005  and  2006,  all  naval  air  stations  transitioned  from  forecasting 
detachments  being  located  on-station  to  having  all  forecasting  duties  performed  via  reach 
back  from  either  FWC  San  Diego  or  FWC  Norfolk.  In  2013,  to  improve  forecasting 
results  during  the  operational  hours  civilian  forecasters  were  employed  at  all  naval  air 
stations  during  the  Monday  through  Friday  operational  hours.  The  1500  UTC  TAF  is 
produced  almost  entirely  by  these  civilian  forecasters.  The  only  exceptions  would  be 
weekends  or  holidays.  During  weekend,  holidays  and  evening  hours,  the  TAFs  were 
continued  to  be  produced  via  reach  back.  This  study  was  made  up  entirely  of  1500  UTC 
TAFs  and  thus  it  reflects  mostly  on-station  forecaster  verification.  AAF  Wheeler  was  not 
used  in  this  analysis  due  to  their  reliance  on  reach  back  for  all  TAFs.  A  small  sample  of 
0700  UTC  data  were  collected  to  compare  the  skill  of  the  reach-back  forecasters  with  the 
on-station  forecasters.  The  0700  UTC  data  collected  were  from  October  through 
December  2013  (Table  18)  and  compared  to  the  same  time  period  for  the  1500  UTC  data. 
When  comparing  the  forecast  accuracy  for  October-December  2013  (Table  19),  the  0700 
UTC  TAF  accuracy  was  1%  better  for  ceiling  and  3%  lower  for  visibility  than  the  1500 
UTC  TAF  accuracy.  The  0700  UTC  LAMP  forecast  accuracy  was  8%  better  for  ceiling 
and  11%  better  for  visibility  than  the  1500  UTC  accuracy. 

When  verifying  the  IMC  forecast,  the  TAF  IMC  forecast  accuracy  at  0700  UTC 
(Table  20)  was  only  1%  lower  for  ceiling  and  equal  for  visibility  compared  to  the  1500 
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UTC  forecasts  (Table  21).  The  accuracy  of  the  0700  UTC  LAMP  was  5%  better  for 
ceiling  and  7%  better  for  visibility  compared  to  the  1500  UTC  LAMP  (Tables  20  and  21). 

For  the  CSI  scores  (Tables  22  through  25),  the  0700  UTC  TAF  slightly 
outperfonned  the  1500  UTC  ceiling  and  visibility  forecasts.  The  0700  UTC  LAMP  also 
outperfonned  the  1500  UTC  LAMP  ceiling  and  visibility  forecasts.  Although  there  is 
only  one  quarter  of  data  used  in  this  portion  of  the  study,  both  the  reach  back  and  on- 
station  TAFs  verifications  were  with  respect  to  the  exact  same  weather  over  the  same 
exact  periods.  Considering  the  numerous  advantages  that  an  on-station  forecaster  has 
over  a  reach  back  forecaster,  it  is  somewhat  surprising  that  the  reach  back  forecaster 
perfonned  as  well  as,  or  even  slightly  better  than  the  on-station  forecaster.  When 
considering  just  these  scores,  the  0700  UTC  LAMP  seems  to  perform  as  better  model 
guidance  than  the  1500  UTC  LAMP.  This  shadowing  of  the  model  perfonnance  could 
lead  to  the  interpretation  that  both  the  on-station  and  reach  back  forecasters  are  relying 
heavily  on  the  model,  and  thus  the  reason  for  the  reach  back  forecasters  better  than 
expected  results  compared  to  the  on-station  forecaster. 

Table  18.  The  0700  UTC  forecast  accuracy  for  categorical  forecasts 
for  all  stations  during  Oct-Dee  2013.  The  number  of 
forecasts  verified  is  listed  for  each  element  of  ceiling  or 
visibility.  The  number  of  correct  forecast  is  listed  in  the 
middle  box. 


OCT-DEC  2013 

0700  UTC 

FORECASTS 

TAF 

OCT-DEC 

2013 

0700  UTC 

FORECASTS 

LAMP 

CEILING 

14345 

11000 

77% 

CEILING 

14345 

11345 

79% 

VISIBILITY 

14298 

10768 

75% 

VISIBILITY 

14298 

11542 

81% 
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Table  19.  The  1500  UTC  forecast  accuracy  for  overall  categorical 

forecasts  for  all  stations  during  Oct-Dec  2013.  The  number 
of  forecasts  verified  is  listed  for  each  element  of  ceiling  or 
visibility.  The  number  of  correct  forecast  is  listed  in  the 
middle  box. 


OCT-DEC  2013 

1500  UTC 

FORECASTS 

TAF 

OCT-DEC 

2013 

1500  UTC 

FORECASTS 

LAMP 

CEILING 

14123 

10787 

76% 

CEILING 

15657 

11058 

71% 

VISIBILITY 

14123 

10981 

78% 

VISIBILITY 

15657 

10891 

69% 

Table  20.  The  0700  UTC  forecast  accuracy  for  IMC  categorical 

forecasts  for  all  stations  during  Oct-Dec  2013.  The  number 
of  forecasts  verified  is  listed  for  each  element  of  ceiling  or 
visibility.  The  number  of  correct  forecast  is  listed  in  the 
middle  box. 


OCT-DEC  2013 

0700  UTC  IMC 

FORECASTS 

TAF 

OCT-DEC  2013 

0700  UTC  IMC 

FORECASTS 

LAMP 

CEILING 

1548 

727 

47% 

CEILING 

1305 

704 

54% 

VISIBILITY 

1570 

466 

30% 

VISIBILITY 

1170 

365 

31% 
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Table  21.  The  1500  UTC  forecast  accuracy  for  IMC  categorical 

forecasts  for  all  stations  during  Oct-Dec  2013.  The  number 
of  forecasts  verified  is  listed  for  each  element  of  ceiling  or 
visibility.  The  number  of  correct  forecast  is  listed  in  the 
middle  box. 


OCT-DEC  2013 

1500  UTC  IMC 

FORECASTS 

TAF 

OCT-DEC  2013 

1500  UTC  IMC 

FORECASTS 

LAMP 

CEILING 

1371 

660 

48% 

CEILING 

1437 

692 

48% 

VISIBILITY 

1250 

385 

31% 

VISIBILITY 

1518 

371 

24% 

Table  22.  The  CSI  for  ceiling  forecasts  made  at  0700  UTC  during 
Oct-Dec  2013 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

727 

735 

IRF 

704 

758 

VFR 

1168 

12062 

VFR 

601 

12282 

CSI 

0.32 

CSI 

0.34 
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Table  23.  The  CSI  for  ceiling  forecasts  made  at  1500  UTC  during 
Oct-Dee  2013 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

660 

752 

IRF 

682 

720 

VFR 

711 

11984 

VFR 

745 

11966 

CSI 

0.31 

CSI 

0.32 

Table  24.  The  CSI  for  visibility  forecasts  made  at  0700  UTC  during 
Oct-Dee  2013 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

466 

378 

IRF 

365 

482 

VFR 

1104 

12350 

VFR 

805 

12693 

CSI 

0.24 

CSI 

0.22 

Table  25.  The  CSI  for  visibility  forecasts  made  at  1500  during  Oct- 
Dee  2013 


TAF 

LAMP 

Forecast 

IFR  Obs 

VFR  Obs 

Forecast 

IFR  Obs 

VFR  Obs 

IFR 

385 

450 

IRF 

371 

471 

VFR 

865 

12261 

VFR 

1147 

12134 

CSI 

0.23 

CSI 

0.19 
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IV.  NAS  FALLON  CASE  STUDY 


A.  OVERVIEW 

There  may  be  countless  instances  where  the  TAF  and  the  LAMP  outperform  one 
another,  and  it  would  be  easy  to  selectively  choose  events  that  made  one  or  the  other 
appear  to  be  significantly  better.  But,  the  aircraft  mishap  on  NAS  Fallon  in  March  6, 
2012  was  a  particularly  significant  event  in  that  it  was  a  fatal  crash  and  the  weather  was 
not  forecast  correctly  in  advance  of  takeoff.  (NTSB,  2014)  The  National  Transportation 
Safety  Board  (NTSB)  did  not  cite  the  poor  weather  as  a  specific  reason  for  the  mishap, 
but  does  list  the  conditions  as  “significantly  lower  than  forecasted”  in  the  contributing 
factors  (NTSB,  2014).  Although  not  the  sole  cause  of  this  accident,  it  can  be  reasonably 
expected  that  a  better  forecast  could  have  deterred  the  pilot  from  taking  off  in  poor 
weather  conditions,  and  thus  possibly  avoided  the  mishap  all  together. 

B.  DATA  FOR  MARCH  6,  2012 

(1)  NAS  Fallon  TAFs 

Figure  18  lists  the  TAFs  issued  for  NAS  Fallon  at  1200  UTC,  1500  UTC,  and 
1700  UTC  6  March  2012.  In  2012,  these  TAFs  were  issued  from  Fleet  Weather  Center 
San  Diego.  Highlighted  in  the  yellow  boxes  are  the  forecast  lines  valid  for  time  of  takeoff 
and  time  of  mishap  for  each  TAF.  The  TAF  used  for  preflight  weather  brief  was  issued 
originally  at  0700  UTC  and  then  amended  at  1200  UTC.  The  conditions  between  1600 
UTC  and  2000  UTC  were  forecast  to  be  no  lower  than  5  miles  visibility  and  4000  feet 
ceilings  in  light  snow.  A  new  TAF  was  routinely  issued  at  1500  UTC,  which  is  a  routine 
forecast  time.  This  1500  UTC  TAF  is  the  valid  forecast  at  the  time  of  scheduled  take  off 
of  1650  UTC.  The  forecast  is  now  unrestricted  visibility  4000  feet  ceilings  throughout  the 
takeoff  and  mishap  time.  At  1720  UTC,  the  TAF  is  amended  to  include  temporary 
conditions  of  0.5  miles  visibility  and  1500  feet  ceilings  in  light  snow.  These  conditions 
are  forecast  on  station  until  2000  UTC. 


41 


(2)  NAS  Fallon  Weather  Observations 

In  the  Appendix,  the  weather  observations  for  1456  UTC  -  2000  UTC  6  March 
2012  at  NAS  Fallon  are  listed.  In  summary,  the  weather  was  VMC  all  day  until  light 
snow  brought  visibility  down  to  1  and  0.5  miles  at  1646  UTC,  and  by  1650  the  visibility 
was  at  0.5  miles  with  ceilings  at  1200  feet.  The  visibility  mostly  stayed  in  IMC  with 
MVMC  ceilings  until  the  snow  stopped  at  1912  UTC. 

(3)  NAS  Fallon  LAMP 

Figure  20  is  the  LAMP  bulletin  for  NAS  Fallon  issued  at  0700  UTC  6  March 
2012.  Although  there  were  LAMP  bulletins  issued  every  hour  prior  to  and  after  the  0700 
UTC  TAF  and  the  1200  UTC  amended  TAF,  this  bulletin  is  a  good  representation  of  the 
model  guidance  at  the  same  time  the  TAF  was  issued.  Highlighted  in  yellow  boxes  are 
the  LAMP  forecasts  for  1800  and  1900  UTC.  Outlined  in  blue,  LAMP  indicates 
conditional  visibility  (CVS)  of  0.5  to  less  than  1  mile  (Code  2)  for  1800  UTC  and  1  to 
less  than  2  miles  (Code  3)  for  1900  UTC,  and  a  conditional  ceiling  (CCG)  of  1500  feet 
(Code  4)  for  both  1800  and  1900  UTC.  Outlined  in  the  orange  is  the  probability  forecast 
of  precipitation  (PPO)  and  the  type  of  precipitation  expected  (TYP).  Snow  was  forecasted 
with  a  probability  of  23%. 

C.  AN  ARGUMENT  FOR  PROBABILITY  FORECASTING 

Although  it  is  not  known  if  the  forecaster  for  the  NAS  Fallon  TAF  was  using  the 
LAMP  bulletin  as  guidance,  it  cannot  be  overlooked  that  the  LAMP  bulletin  gave  at  least 
a  hint  of  IMC  visibility  and  lower  MVMC  ceilings  during  the  flight  mission  window. 
According  to  the  NTSB  report,  just  prior  to  take  off  at  1645  UTC,  the  pilot  radioed  the 
duty  observer  for  a  weather  update  when  conditions  deteriorated  below  what  was 
originally  forecasted.  It  is  possible  that  if  the  pilot  had  been  briefed  of  even  a  23%  chance 
of  IMC  conditions,  he  may  not  have  decided  to  go  forward  with  the  mission  once  he  saw 
the  conditions  deteriorating  in  front  of  him.  Although  23%  seems  like  conditions  are 
more  likely  to  not  occur  than  to  occur,  it  is  still  a  hint  that  something  has  changed  in  the 
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atmosphere  and  is  thus  providing  a  sense  of  some  change  more  than  a  TAF  using 
detenninistic  forecasting. 

It  should  be  noted  that  the  LAMP  guidance  was  not  a  perfect  forecast  for  this 
event.  For  the  same  day,  the  1400  UTC  LAMP  bulletin  (Figure  21),  which  would  have 
been  the  guidance  for  the  1500  UTC  TAF,  decreased  the  probability  of  precipitation  from 
23%  to  14%  and  slightly  raised  the  visibility  one  code  value  to  1  to  less  than  2  miles 
(Code  3)  at  1800  UTC  and  2  less  than  3  miles  (Code  4)  at  1900  UTC.  It  also  eliminated 
the  ceilings  and  went  to  scattered  skies.  So  although  the  model  cannot  be  used,  if 
properly  studied,  tendencies  in  the  model’s  probability  forecasts  can  be  tracked  and  later 
used  to  develop  more  accurate  forecasting  guidance. 
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Figure  16.  NAS  Fallon  TAFs  issued  at  1200,  1500,  and  1700  UTC  for  6  March, 
2012. 


201 203061 300  TAF  AMD  KNFL  0612/0707 2401 5G24KT9999  SCT040  SCT1 30  BKN200  640409 
641308  520009  520909  541809  522709  523604 
QNH2960INS 

BECMG  061 2/061 4  3201 8G27KfBOOO-SN  SCT020  BKN040  I 
OVC060  640409  641308  520009  520909 
541809  522709  523604  QNH2955INS 
TEMPO  0614/0620  33022G32KT 
BECMG  0621/0623  34025G35KT  9000  -SHSN  FEW020 
SCT040  BKN060  BKN100  BKN200  640409  641 308 
520009  520909  541809  522709  523604  QNH2960INS 
BECMG  0702/0704  35025G35KT  9999  VCSH  SCT040 
BKN060  BKN100  640409  641308  520009  520909 
541809  522709  523604  QNH2970INS 
TEMPO  0704/0707  34030G38KT  9000  BLDU  FEW040 

SCT060  SCT1 00  T09/061 4Z  TM02/0706Z  AMD  1 252= _ 


201 203061 529  TAF  KNFL  061 5/071 5 3201 8G27KT  9999  VCSH  SCT020  BKN040  OVC060  140409 

641308  540009  540909  541809  522709  523BU4 - 

QNH2955INS 

TEMPO  061 5/0620  33022G32KT 
BECMG  0621/0623  34025G35KT  9000  -SHSN  FEW020 
SCT040  BKN060  BKN1 00  BKN200  640409  641 308 
540009  540909  541 809  522709  523604  QNH2960INS 
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Figure  17.  NAS  Fallon  LAMP  bulletin  issued  0700  UTC  6  March  2012. 
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Figure  18.  NAS  Fallon  LAMP  bulletin  issued  0700  UTC  6  March  2012. 
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V.  CONCLUSION 


A.  SUMMARY 

The  observational  statistics  indicate  that  for  the  stations  chosen  in  this  study  the 
weather  conditions  are  above  IMC  over  94%  of  the  time.  Both  the  forecaster  and  the 
model  correctly  verify  those  conditions  over  95%  of  the  time.  This  is  a  clear  indication 
that  fair-weather  not  only  occurs  frequently,  but  is  fairly  easy  to  predict.  The  IMC  events 
occur  only  occurring  6%  of  the  time.  Unfortunately,  both  the  model  and  the  forecaster 
have  only  about  a  47%  hit  rate  for  the  ceiling  IMC  events  and  43%  hit  rate  for  visibility 
IMC  events  (Figure  22)  .  Since  the  occurrence  of  IMC  conditions  is  when  the  customers 
are  predominantly  relying  on  the  forecaster,  the  forecast  accuracy  during  those  instances 
becomes  the  primary  source  of  the  customer’s  judgment  of  a  forecaster’s  value,  despite 
the  low  frequency  of  occurrence. 

When  the  full  24  hr  forecast  is  compared  to  the  13  through  24  h  forecasts,  there 
are  no  significant  differences  with  the  00-12  h  forecasts.  But,  when  the  October  2006 
through  March  2007  data  are  compared  to  the  same  months  in  2013,  there  is  a  significant 
improvement  in  skill  across  for  both  the  model  and  the  forecaster.  These  improvements 
can  likely  be  attributed  to  the  improved  accuracy  in  model  forecasts.  However,  skill  with 
respect  to  visibility  forecasts  by  the  forecaster  improved  at  a  significantly  higher  rate  than 
the  model  did  during  this  period.  This  might  be  attributed  to  the  change  of  reach  back 
forecasting  form  2006  through  2012,  to  the  on  station  forecasts  produced  in  2013. 

Using  CSI  as  the  main  indicator  of  skill,  there  is  very  little  difference  between  the 
model  and  the  TAF.  Even  when  using  the  Non-VMC  HR  and  CSI,  both  the  model  and 
TAF  performed  equally  well.  However,  the  forecaster  did  not  improve  upon  the  model’s 
perfonnance.  Therefore,  this  study  could  not  detennine  any  significant  value  added  by 
the  forecaster  over  the  model  in  2013.  As  for  whether  those  CSI  scores  are  overall  good 
or  poor  is  difficult  to  detennine.  However,  the  most  recent  verification  that  could  be 
found  from  2008  (Figures  23  and  24),  shows  LAMP  with  an  average  CSI  of  .36  for 
ceilings  and  an  average  of  .28  for  visibility.  Although  these  numbers  are  higher,  they  are 
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consistent  with  what  this  study  has  calculated  and  therefore  provide  some  legitimacy  into 
the  methods  of  verification. 


Figure  19.  2013  Observational  hours  VMC  vs  IMC  (all  stations) 
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Figure  20.  Verification  of  ceiling  IMC  forecasts  by  LAMP  during  October  2007 
March  2008  (from  Rudack  and  Ghirardelli,  2010) 


Figure  2 1 .  Verification  of  ceiling  IMC  forecasts  by  LAMP  during  October  2007 
March  2008  (from  Rudack  and  Ghirardelli  2010) 
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B.  RECOMMENDATIONS  FOR  EXPLOITING  MODEL  STRENGTH 

(1)  Probabilistic  Forecasting 

As  shown  in  the  case  study,  using  probabilistic  forecasting  techniques  has 
potential  to  be  a  better  decision  aid  than  a  deterministic  forecast.  At  the  very  least,  the 
forecasters  can  track  the  models  probability  forecasts  and  develop  rules  of  thumb  to 
incorporate  into  their  TAF  writing  as  a  measure  of  the  forecast  utility.  For  example,  a 
23%  chance  of  precipitation  at  NAS  Fallon,  might  weigh  differently  than  a  23%  chance 
of  precipitation  at  NAS  Oceana.  But  ideally,  the  Navy  will  leam  to  use  probabilistic 
forecasting  in  the  TAF  and  other  forecasting  products.  This  will  provide  the  customer 
more  information  into  what  the  forecaster  is  expecting. 

(2)  Model  Interpreters 

With  the  lack  of  hands  on  training  available  in  the  fleet  compared  to  twenty  years 
ago,  and  the  ever  increasing  shift  to  have  Aerographer’s  Mates  perform  non-forecasting 
jobs,  it  seems  more  productive  to  have  them  become  more  of  a  model  interpreter  than  a 
forecaster.  More  model  training  needs  to  be  added  to  the  base  level  forecasting  school 
and  incorporated  into  the  training  pipeline  in  order  for  them  to  understand  proper 
techniques  to  use  the  model  more  adequately.  Case  studies  tracking  model  performance 
in  various  environments  should  be  a  part  of  the  daily  routine,  or  at  least  during  the  94% 
of  shift  time  that  is  VMC.  Cataloging  these  model  perfonnances  will  lead  to  better  “rules 
of  thumb”  for  forecasters  to  access  when  faced  with  time  sensitive  forecasting,  or  in 
meteorological  events  that  they  have  not  experienced  during  their  tour. 

(3)  Standardized  Verification  Program 

The  Navy  did  not  perform  verification  of  its  TAFs  in  2013.  In  combination  with 
training,  verification  is  the  primary  method  to  improve  on  forecasting  skill.  The  National 
Weather  Service  is  currently  refitting  their  online  verification  tool  to  be  compatible  with 
the  Navy  TAFs.  Once  online,  the  verification  program  should  be  standardized  across  all 
stations  and  overseen  at  the  CNMOC  level.  Oversight  and  standardization  will  ensure 
that  forecasting  weaknesses  will  be  noticed  earlier  and  incorporated  into  the  training 
process  across  the  fleet. 
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C.  RECOMMENDATIONS  FOR  FUTURE  STUDIES 

(1)  Model  tendencies 

Run  a  verification  program  to  grade  the  skill  of  LAMPs  probability  forecasts  to 
identify  if  there  are  any  tendencies  to  be  exploited  by  forecasters  at  each  station. 

(2)  Thunderstorm  and  wind  data 

Thunderstorm  and  high  wind  condition  may  not  always  be  associated  with  an 
IMC  event.  Yet  they  both  have  the  potential  to  impact  air  operations  equally.  The  TAF 
and  LAMP  bulletin  for  thunderstorm  and  wind  data  should  be  verified  to  detennine  the 
level  of  value  the  forecaster  is  providing  during  these  two  meteorological  events. 

(3)  Trends 

For  the  possibility  of  predictable  trends  that  may  occur  in  the  model  guidance 
should  be  examined.  If  trends  exist,  they  can  be  exploited  to  define  the  utility  associated 
with  probabilistic  forecasts. 
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APPENDIX 


201203061456  METAR  KNFL  061456Z  26019G24KT  10SM  FEW060  SCT080  BKN120  06/M02 
A2965  RMK  A02  SLP025  ACSL  DSNT  NW 
T00561017 56004= 

201203061556  METAR  KNFL  061556Z  28021G25KT  10SM  FEW050  BKN075  BKN120  07/M02 
A2967  RMK  A02  PK  WND  26026/1546  SLP030 
SH  DSNT  W-NW  AND  NE  T00671022= 

201203061636  SPECI  KNFL  061636Z  33021G33KT  10SM  -SN  BKN050  BKN065  OVC120 
03/M03  A2969  RMK  A02  PK  WND  33033/1631 
SNB36  P0000= 

201203061636  SPECI  KNFL  061636Z  33021G33KT  10SM  -SN  BKN050  BKN065  OVC120 
03/M03  A2969  RMK  A02  PK  WND  33033/1631 
SNB36  P0000= 

201203061646  SPECI  KNFL  061646Z  34021G31KT  1  1/2SM  -SN  BKN033  BKN047  OVC085 
02/M02  A2970  RMK  A02  PK  WND  33033/1631 
SFC  VIS  2  1/2  SNB36  RCRNR  P0000= 

201203061646  SPECI  KNFL  061646Z  34021G31KT  1  1/2SM  -SN  BKN033  BKN047  OVC085 
02/M02  A2970  RMK  A02  PK  WND  33033/1631 
SFC  VIS  2  1/2  SNB36  RCRNR  P0000= 

201203061650  SPECI  KNFL  061650Z  34021G33KT  1/2SM  SN  OVC012  01/M02  A2970 
RMK  A02  PK  WND  35033/1648  SNB36  RCRNR 
P0000= 

201203061650  SPECI  KNFL  061650Z  34021G33KT  1/2SM  SN  OVC012  01/M02  A2970 
RMK  A02  PK  WND  35033/1648  SNB36  RCRNR 
P0000= 

201203061656  METAR  KNFL  061656Z  35024G33KT  1/2SM  SN  FZFG  OVC012  M01/M03 
A2971  RMK  A02  PK  WND  35033/1654  SNB36 
SLP041  RCRNR  POOOO  T10061028= 

201203061703  SPECI  KNFL  061703Z  35022G33KT  1/2SM  -SN  BKN015  OVC045  M01/M03 
A2971  RMK  A02  PK  WND  36031/1701  SFC 
VIS  3/4  RNRNR  P0000= 

201203061703  SPECI  KNFL  061703Z  35022G33KT  1/2SM  -SN  BKN015  OVC045  M01/M03 
A2971  RMK  A02  PK  WND  36031/1701  SFC 
VIS  3/4  RNRNR  P0000= 

201203061720  SPECI  KNFL  061720Z  35023G34KT  1  1/2SM  -SN  BKN015  OVC045  M01/M04 
A2972  RMK  A02  PK  WND  36041/1705  SFC 
VIS  2  WR//  P0000= 

201203061720  SPECI  KNFL  061720Z  35023G34KT  1  1/2SM  -SN  BKN015  OVC045  M01/M04 
A2972  RMK  A02  PK  WND  36041/1705  SFC 
VIS  2  WR//  P0000= 

201203061727  SPECI  KNFL  061727Z  36021G34KT  1  1/2SM  -SN  BKN022  BKN042  OVC050 
M01/M04  A2973  RMK  A02  PK  WND  36041/1705 
SFC  VIS  2  WR//  P0000= 

201203061727  SPECI  KNFL  061727Z  36021G34KT  1  1/2SM  -SN  BKN022  BKN042  OVC050 
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M01/M04  A2973  RMK  AO 2  PK  WND  36041/1705 
SFC  VIS  2  WR//  P0000= 

201203061734  SPECI  KNFL  061734Z  36028G33KT  2SM  -SN  BKN030  OVC045  M01/M05 
A2973  RMK  A02  PK  WND  36041/1705  SFC 
VIS  3  WR//  P0000= 

201203061734  SPECI  KNFL  061734Z  36028G33KT  2SM  -SN  BKN030  OVC045  M01/M05 
A2973  RMK  A02  PK  WND  36041/1705  SFC 
VIS  3  WR//  P0000= 

201203061748  SPECI  KNFL  061748Z  35021G29KT  ISM  -SN  OVC025  M01/M04  A2976 
RMK  AO 2  PK  WND  36041/1705  TWR  VIS 
1  1/2  WR//  P0000= 

201203061748  SPECI  KNFL  061748Z  35021G29KT  ISM  -SN  OVC025  M01/M04  A2976 
RMK  AO 2  PK  WND  36041/1705  TWR  VIS 
1  1/2  WR//  P0000= 

201203061756  METAR  KNFL  061756Z  36019G29KT  3SM  -SN  BR  SCT025  BKN034  OVC041 
M02/M04  A2976  RMK  A02  PK  WND  36041/1705 
SLP056  WR//  P0000  60000  T10171039 
10094  21017  53031= 

201203061822  SPECI  KNFL  061822Z  34023G28KT  2SM  -SN  OVC027  M01/M04  A2978 
RMK  A02  PK  WND  34030/1812  WR//  P0000= 

201203061822  SPECI  KNFL  061822Z  34023G28KT  2SM  -SN  OVC027  M01/M04  A2978 
RMK  A02  PK  WND  34030/1812  WR//  P0000= 

201203061836  SPECI  KNFL  061836Z  34026G33KT  3SM  -SN  BKN023  BKN028  OVC040 
M01/M05  A2979  RMK  A02  PK  WND  35033/1836 
WR//  P0000= 

201203061836  SPECI  KNFL  061836Z  34026G33KT  3SM  -SN  BKN023  BKN028  OVC040 
M01/M05  A2979  RMK  A02  PK  WND  35033/1836 
WR//  P0000= 

201203061850  SPECI  KNFL061850Z  34022G34KT  3/4SM  -SN  BKN025  OVC037  M01/M06 
A2980  RMK  A02  PK  WND  34034/1843  SFC 
VIS  1  WR//  P0000= 

201203061850  SPECI  KNFL061850Z  34022G34KT  3/4SM  -SN  BKN025  OVC037  M01/M06 
A2980  RMK  A02  PK  WND  34034/1843  SFC 
VIS  1  WR//  P0000= 

201203061852  SPECI  KNFL  061852Z  34020G34KT  3/4SM  -SN  BKN022  BKN028  OVC037 
M01/M06  A2981  RMK  A02  PK  WND  34034/1843 
SFC  VIS  1  WR//  P0000= 

201203061852  SPECI  KNFL  061852Z  34020G34KT  3/4SM  -SN  BKN022  BKN028  OVC037 
M01/M06  A2981  RMK  A02  PK  WND  34034/1843 
SFC  VIS  1  WR//  P0000= 

201203061856  METAR  KNFL  061856Z  35025G31KT  3/4SM  -SN  BKN020  BKN028  OVC041 
M01/M05  A2981  RMK  A02  PK  WND  34034/1843 
SFC  VIS  1  SLP072  WR//  P0000  T10111050= 

201203061903  SPECI  KNFL  061903Z  34022G34KT  1  1/2SM  -SN  BKN018  BKN026  OVC041 
M01/M05  A2981  RMK  A02  PK  WND  35034/1858 
SFC  VIS  2  WR//  P0000= 

201203061903  SPECI  KNFL  061903Z  34022G34KT  1  1/2SM  -SN  BKN018  BKN026  OVC041 
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M01/M05  A2981  RMK  AO 2  PK  WND  35034/1858 
SFC  VIS  2  WR//  P0000= 

201203061912  SPECI  KNFL  061912Z  COR  34018G30KT  7SM  BKN020  BKN030  BKN037 
M01/M05  A2982  RMK  A02  PK  WND  35034/1858 
SH  VC  E-S-SW  SNE12  WR//  P0000= 

201203061912  SPECI  KNFL  061912Z  COR  34018G30KT  7SM  BKN020  BKN030  BKN037 
M01/M05  A2982  RMK  A02  PK  WND  35034/1858 
SH  VC  E-S-SW  SNE12  WR//  P0000= 

201203061947  SPECI  KNFL  061947Z  35020G25KT  10SM  BKN020  BKN030  BKN037  01/M06 
A2982  RMK  A02  PK  WND  35034/1858  SNE12 
WR//  P0000= 

201203061947  SPECI  KNFL  061947Z  35020G25KT  10SM  BKN020  BKN030  BKN037  01/M06 
A2982  RMK  A02  PK  WND  35034/1858  SNE12 
WR//  P0000= 

201203061956  METAR  KNFL  061956Z  35021G29KT  8SM  -SN  SCT030  BKN037  BKN070 
01/M06  A2982  RMK  A02  PK  WND  35034/1858 
SLP076  SNE12  SH  VC  N-E-SE  P0000  T00061061= 

201203062004  SPECI  KNFL  062004Z  01019G30KT  10SM  -SN  SCT030  BKN040  BKN070 
OVC200  01/M06  A2982  RMK  A02  PK  WND 
36030/2001  SNB01  VIS  LWR  NE-E  P0000= 
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